Data Lineage: A Survey

Lineage, or provenance, in its most general form describes where data came from, how it was derived, and how it was updated over time. Information management systems today exploit lineage in tasks ranging from data verification in curated databases [1] to confidence computation in probabilistic databases [10, 12]. Here, we formalize and categorize lineage, discuss a set of selected papers, and then identify open problems in lineage research. Lineage can be useful in a variety of settings. For example, molecular biology databases, which mostly store copied data, can use lineage to verify the copied data by tracking the original sources [1]. Data warehouses can use the lineage of anomalous view data to identify faulty base data [4], and probabilistic databases can exploit lineage for confidence computation [10, 12]. Although lineage can be very valuable for applications, storing and querying lineage can be expensive; for instance the Gene Ontology database has up to 10MB of lineage for single tuples [10]. Approaches have been developed to lower these costs. For example, in probabilistic databases, approximate lineage can compress the complete lineage by up to two orders of magnitude while allowing a selected set of queries over the lineage to be answered efficiently with low error [10]. In curated databases, storing a type of lineage called hierarchical transactional provenance can reduce the storage overhead by a factor of 5, relative to a more naive approach [1]. In addition to challenges related to space and time efficiency, it can be difficult even to define lineage in domains that allow arbitrary transformations. The most commonly considered transformations are relational queries [4, 9, 10, 12], but some papers have studied lineage for a broad range of transformations. For example, in data warehousing, tracing procedures have been developed for general transformations that take advantage of transformation properties specified by the transformation definer [3]. In curated databases, lineage for copying operations has been studied [1]. For a generalization of standard relational queries called DQL (Deterministic QL), definitions for lineage that are invariant under query rewriting have been proposed [2].